{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<style>\n",
       "@import url('http://fonts.googleapis.com/css?family=Source+Code+Pro');\n",
       "@import url('http://fonts.googleapis.com/css?family=Vollkorn');\n",
       "@import url('http://fonts.googleapis.com/css?family=Arimo');\n",
       "@import url('http://fonts.googleapis.com/css?family=Fira_sans');\n",
       "\n",
       "    div.cell{\n",
       "        width: 900px;\n",
       "        margin-left: 0% !important;\n",
       "        margin-right: auto;\n",
       "    }\n",
       "    div.text_cell code {\n",
       "        background: transparent;\n",
       "        color: #000000;\n",
       "        font-weight: 600;\n",
       "        font-size: 11pt;\n",
       "        font-style: bold;\n",
       "        font-family:  'Source Code Pro', Consolas, monocco, monospace;\n",
       "   }\n",
       "    h1 {\n",
       "        font-family: 'Open sans',verdana,arial,sans-serif;\n",
       "\t}\n",
       "\t\n",
       "    div.input_area {\n",
       "        background: #F6F6F9;\n",
       "        border: 1px solid #586e75;\n",
       "    }\n",
       "\n",
       "    .text_cell_render h1 {\n",
       "        font-weight: 200;\n",
       "        font-size: 30pt;\n",
       "        line-height: 100%;\n",
       "        color:#c76c0c;\n",
       "        margin-bottom: 0.5em;\n",
       "        margin-top: 1em;\n",
       "        display: block;\n",
       "        white-space: wrap;\n",
       "        text-align: left;\n",
       "    } \n",
       "    h2 {\n",
       "        font-family: 'Open sans',verdana,arial,sans-serif;\n",
       "        text-align: left;\n",
       "    }\n",
       "    .text_cell_render h2 {\n",
       "        font-weight: 200;\n",
       "        font-size: 16pt;\n",
       "        font-style: italic;\n",
       "        line-height: 100%;\n",
       "        color:#c76c0c;\n",
       "        margin-bottom: 0.5em;\n",
       "        margin-top: 1.5em;\n",
       "        display: block;\n",
       "        white-space: wrap;\n",
       "        text-align: left;\n",
       "    } \n",
       "    h3 {\n",
       "        font-family: 'Open sans',verdana,arial,sans-serif;\n",
       "    }\n",
       "    .text_cell_render h3 {\n",
       "        font-weight: 200;\n",
       "        font-size: 14pt;\n",
       "        line-height: 100%;\n",
       "        color:#d77c0c;\n",
       "        margin-bottom: 0.5em;\n",
       "        margin-top: 2em;\n",
       "        display: block;\n",
       "        white-space: wrap;\n",
       "        text-align: left;\n",
       "    }\n",
       "    h4 {\n",
       "        font-family: 'Open sans',verdana,arial,sans-serif;\n",
       "    }\n",
       "    .text_cell_render h4 {\n",
       "        font-weight: 100;\n",
       "        font-size: 14pt;\n",
       "        color:#d77c0c;\n",
       "        margin-bottom: 0.5em;\n",
       "        margin-top: 0.5em;\n",
       "        display: block;\n",
       "        white-space: nowrap;\n",
       "    }\n",
       "    h5 {\n",
       "        font-family: 'Open sans',verdana,arial,sans-serif;\n",
       "    }\n",
       "    .text_cell_render h5 {\n",
       "        font-weight: 200;\n",
       "        font-style: normal;\n",
       "        color: #1d3b84;\n",
       "        font-size: 16pt;\n",
       "        margin-bottom: 0em;\n",
       "        margin-top: 0.5em;\n",
       "        display: block;\n",
       "        white-space: nowrap;\n",
       "    }\n",
       "    div.text_cell_render{\n",
       "        font-family: 'Fira sans', verdana,arial,sans-serif;\n",
       "        line-height: 125%;\n",
       "        font-size: 115%;\n",
       "        text-align:justify;\n",
       "        text-justify:inter-word;\n",
       "    }\n",
       "    div.output_subarea.output_text.output_pyout {\n",
       "        overflow-x: auto;\n",
       "        overflow-y: scroll;\n",
       "        max-height: 50000px;\n",
       "    }\n",
       "    div.output_subarea.output_stream.output_stdout.output_text {\n",
       "        overflow-x: auto;\n",
       "        overflow-y: scroll;\n",
       "        max-height: 50000px;\n",
       "    }\n",
       "    div.output_wrapper{\n",
       "        margin-top:0.2em;\n",
       "        margin-bottom:0.2em;\n",
       "}\n",
       "\n",
       "    code{\n",
       "      font-size: 70%;\n",
       "    }\n",
       "    .rendered_html code{\n",
       "    background-color: transparent;\n",
       "    }\n",
       "    ul{\n",
       "        margin: 2em;\n",
       "    }\n",
       "    ul li{\n",
       "        padding-left: 0.5em; \n",
       "        margin-bottom: 0.5em; \n",
       "        margin-top: 0.5em; \n",
       "    }\n",
       "    ul li li{\n",
       "        padding-left: 0.2em; \n",
       "        margin-bottom: 0.2em; \n",
       "        margin-top: 0.2em; \n",
       "    }\n",
       "    ol{\n",
       "        margin: 2em;\n",
       "    }\n",
       "    ol li{\n",
       "        padding-left: 0.5em; \n",
       "        margin-bottom: 0.5em; \n",
       "        margin-top: 0.5em; \n",
       "    }\n",
       "    ul li{\n",
       "        padding-left: 0.5em; \n",
       "        margin-bottom: 0.5em; \n",
       "        margin-top: 0.2em; \n",
       "    }\n",
       "    a:link{\n",
       "       font-weight: bold;\n",
       "       color:#447adb;\n",
       "    }\n",
       "    a:visited{\n",
       "       font-weight: bold;\n",
       "       color: #1d3b84;\n",
       "    }\n",
       "    a:hover{\n",
       "       font-weight: bold;\n",
       "       color: #1d3b84;\n",
       "    }\n",
       "    a:focus{\n",
       "       font-weight: bold;\n",
       "       color:#447adb;\n",
       "    }\n",
       "    a:active{\n",
       "       font-weight: bold;\n",
       "       color:#447adb;\n",
       "    }\n",
       "    .rendered_html :link {\n",
       "       text-decoration: underline; \n",
       "    }\n",
       "    .rendered_html :hover {\n",
       "       text-decoration: none; \n",
       "    }\n",
       "    .rendered_html :visited {\n",
       "      text-decoration: none;\n",
       "    }\n",
       "    .rendered_html :focus {\n",
       "      text-decoration: none;\n",
       "    }\n",
       "    .rendered_html :active {\n",
       "      text-decoration: none;\n",
       "    }\n",
       "    .warning{\n",
       "        color: rgb( 240, 20, 20 )\n",
       "    } \n",
       "    hr {\n",
       "      color: #f3f3f3;\n",
       "      background-color: #f3f3f3;\n",
       "      height: 1px;\n",
       "    }\n",
       "    blockquote{\n",
       "      display:block;\n",
       "      background: #fcfcfc;\n",
       "      border-left: 5px solid #c76c0c;\n",
       "      font-family: 'Open sans',verdana,arial,sans-serif;\n",
       "      width:680px;\n",
       "      padding: 10px 10px 10px 10px;\n",
       "      text-align:justify;\n",
       "      text-justify:inter-word;\n",
       "      }\n",
       "      blockquote p {\n",
       "        margin-bottom: 0;\n",
       "        line-height: 125%;\n",
       "        font-size: 100%;\n",
       "      }\n",
       "</style>\n",
       "<script>\n",
       "    MathJax.Hub.Config({\n",
       "                        TeX: {\n",
       "                           extensions: [\"AMSmath.js\"]\n",
       "                           },\n",
       "                tex2jax: {\n",
       "                    inlineMath: [ ['$','$'], [\"\\\\(\",\"\\\\)\"] ],\n",
       "                    displayMath: [ ['$$','$$'], [\"\\\\[\",\"\\\\]\"] ]\n",
       "                },\n",
       "                displayAlign: 'center', // Change this to 'center' to center equations.\n",
       "                \"HTML-CSS\": {\n",
       "                    scale:100,\n",
       "                        availableFonts: [],\n",
       "                        preferredFont:null,\n",
       "                        webFont: \"TeX\",\n",
       "                    styles: {'.MathJax_Display': {\"margin\": 4}}\n",
       "                }\n",
       "        });\n",
       "</script>\n"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "execution_count": 1,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# format the book\n",
    "%matplotlib inline\n",
    "import sys\n",
    "from __future__ import division, print_function\n",
    "import sys\n",
    "sys.path.insert(0,'../code')\n",
    "import book_format\n",
    "book_format.load_style('../code')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Hypothesis Testing\n",
    "\n",
    "## Back to the Euro problem\n",
    "\n",
    "In Section [euro] I presented a problem from MacKay’s `*Information\n",
    "Theory, Inference, and Learning Algorithms*`:\n",
    "\n",
    "> A statistical statement appeared in \\`\\`The Guardian\" on Friday\n",
    "> January 4, 2002:\n",
    ">\n",
    "> > When spun on edge 250 times, a Belgian one-euro coin came up heads\n",
    "> > 140 times and tails 110. ‘It looks very suspicious to me,’ said\n",
    "> > Barry Blight, a statistics lecturer at the London School of\n",
    "> > Economics. ‘If the coin were unbiased, the chance of getting a\n",
    "> > result as extreme as that would be less than 7%.’\n",
    ">\n",
    "> But do these data give evidence that the coin is biased rather than\n",
    "> fair?\n",
    "\n",
    "We estimated the probability that the coin would land face up, but we\n",
    "didn’t really answer MacKay’s question: Do the data give evidence that\n",
    "the coin is biased?\n",
    "\n",
    "In Chapter [more] I proposed that data are in favor of a hypothesis if\n",
    "the data are more likely under the hypothesis than under the alternative\n",
    "or, equivalently, if the Bayes factor is greater than 1.\n",
    "\n",
    "In the Euro example, we have two hypotheses to consider: I’ll use $F$\n",
    "for the hypothesis that the coin is fair and $B$ for the hypothesis that\n",
    "it is biased.\n",
    "\n",
    "If the coin is fair, it is easy to compute the likelihood of the data,\n",
    "$\\mathrm{p}(D|F)$. In fact, we already wrote the function\n",
    "that does it.\n",
    "\n",
    "```python\n",
    "def Likelihood(self, data, hypo):\n",
    "    x = hypo / 100.0\n",
    "    head, tails = data\n",
    "    like = x**heads * (1-x)**tails\n",
    "    return like\n",
    "```\n",
    "\n",
    "To use it we can create a `Euro` suite and invoke\n",
    "`Likelihood`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "import euro\n",
    "from euro import Euro\n",
    "\n",
    "data = 'H' * 140 + 'T' * 110\n",
    "\n",
    "suite = Euro()\n",
    "likelihood = suite.Likelihood(data, 50)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$\\mathrm{p}(D|F)$ is $5.5 \\cdot 10^{-76}$, which doesn’t\n",
    "tell us much except that the probability of seeing any particular\n",
    "dataset is very small. It takes two likelihoods to make a ratio, so we\n",
    "also have to compute $\\mathrm{p}(D|B)$`.\n",
    "\n",
    "It is not obvious how to compute the likelihood of $B$, because it’s not\n",
    "obvious what “biased” means.\n",
    "\n",
    "One possibility is to cheat and look at the data before we define the\n",
    "hypothesis. In that case we would say that “biased” means that the\n",
    "probability of heads is 140/250."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "actual_percent = 100.0 * 140 / 250\n",
    "likelihood = suite.Likelihood(data, actual_percent)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This version of $B$ I call `B_cheat`; the likelihood of `b_cheat` is\n",
    "$34 \\cdot 10^{-76}$ and the likelihood ratio is 6.1. So we would say\n",
    "that the data are evidence in favor of this version of $B$.\n",
    "\n",
    "But using the data to formulate the hypothesis is obviously bogus. By\n",
    "that definition, any dataset would be evidence in favor of $B$, unless\n",
    "the observed percentage of heads is exactly 50%."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Making a fair comparison \n",
    "\n",
    "To make a legitimate comparison, we have to define $B$ without looking\n",
    "at the data. So let’s try a different definition. If you inspect a\n",
    "Belgian Euro coin, you might notice that the “heads” side is more\n",
    "prominent than the “tails” side. You might expect the shape to have some\n",
    "effect on $x$, but be unsure whether it makes heads more or less likely.\n",
    "So you might say “I think the coin is biased so that $x$ is either 0.6\n",
    "or 0.4, but I am not sure which.”\n",
    "\n",
    "We can think of this version, which I’ll call `B_two` as a hypothesis\n",
    "made up of two sub-hypotheses. We can compute the likelihood for each\n",
    "sub-hypothesis and then compute the average likelihood."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "like40 = suite.Likelihood(data, 40)\n",
    "like60 = suite.Likelihood(data, 60)\n",
    "likelihood = 0.5 * like40 + 0.5 * like60"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The likelihood ratio (or Bayes factor) for `b_two` is 1.3, which means\n",
    "the data provide weak evidence in favor of `b_two`.\n",
    "\n",
    "More generally, suppose you suspect that the coin is biased, but you\n",
    "have no clue about the value of $x$. In that case you might build a\n",
    "Suite, which I call `b_uniform`, to represent sub-hypotheses from 0 to\n",
    "100."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.9900990099009908"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "b_uniform = Euro(range(0, 101))\n",
    "b_uniform.Remove(50)\n",
    "b_uniform.Normalize()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "I initialize `b_uniform` with values from 0 to 100. I removed the\n",
    "sub-hypothesis that $x$ is 50%, because if $x$ is 50% the coin is fair,\n",
    "but it has almost no effect on the result whether you remove it or not.\n",
    "\n",
    "To compute the likelihood of `b_uniform` we compute the likelihood of\n",
    "each sub-hypothesis and accumulate a weighted average."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def SuiteLikelihood(suite, data):\n",
    "    total = 0\n",
    "    for hypo, prob in suite.Items():\n",
    "        like = suite.Likelihood(data, hypo)\n",
    "        total += prob * like\n",
    "    return total"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The likelihood ratio for `b_uniform` is 0.47, which means that the data\n",
    "are weak evidence against `b_uniform`, compared to $F$.\n",
    "\n",
    "If you think about the computation performed by `SuiteLikelihood`, you\n",
    "might notice that it is similar to an update. To refresh your memory,\n",
    "here’s the `Update` function:\n",
    "\n",
    "```python\n",
    "def Update(self, data):\n",
    "    for hypo in self.Values():\n",
    "        like = self.Likelihood(data, hypo)\n",
    "        self.Mult(hypo, like)\n",
    "    return self.Normalize()\n",
    "```\n",
    "\n",
    "And here’s `Normalize`:\n",
    "\n",
    "```python\n",
    "\n",
    "def Normalize(self):\n",
    "    total = self.Total()\n",
    "\n",
    "    factor = 1.0 / total\n",
    "    for x in self.d:\n",
    "        self.d[x] *= factor\n",
    "\n",
    "    return total\n",
    "```\n",
    "\n",
    "The return value from `Normalize` is the total of the\n",
    "probabilities in the Suite, which is the average of the likelihoods for\n",
    "the sub-hypotheses, weighted by the prior probabilities. And\n",
    "`Update` passes this value along, so instead of using\n",
    "`SuiteLikelihood`, we could compute the likelihood of\n",
    "`b_uniform` like this:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.4999999999999996"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "likelihood = b_uniform.Update(data)\n",
    "likelihood"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## The triangle prior\n",
    "\n",
    "In Chapter [more] we also considered a triangle-shaped prior that gives\n",
    "higher probability to values of $x$ near 50%. If we think of this prior\n",
    "as a suite of sub-hypotheses, we can compute its likelihood like this:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "from thinkbayes import Suite\n",
    "\n",
    "def TrianglePrior(suite):\n",
    "    for x in range(0, 51):\n",
    "        suite.Set(x, x)\n",
    "    for x in range(51, 101):\n",
    "        suite.Set(x, 100-x) \n",
    "    suite.Normalize()\n",
    "    return suite"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.49999999999999994"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "suite = Euro(range(0, 101))\n",
    "\n",
    "b_triangle = TrianglePrior(suite)\n",
    "likelihood = b_triangle.Update(data)\n",
    "likelihood"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The likelihood ratio for `b_triangle` is 0.84, compared to $F$, so again\n",
    "we would say that the data are weak evidence against $B$.\n",
    "\n",
    "The following table shows the priors we have considered, the likelihood\n",
    "of each, and the likelihood ratio (or Bayes factor) relative to $F$.\n",
    "\n",
    "      -------------- ------------------- --------\n",
    "      Hypothesis              Likelihood    Bayes\n",
    "                       $\\times 10^{-76}$   Factor\n",
    "      $F$                            5.5        –\n",
    "      `B_cheat`                       34      6.1\n",
    "      `B_two`                        7.4      1.3\n",
    "      `B_uniform`                    2.6     0.47\n",
    "      `B_triangle`                   4.6     0.84\n",
    "      -------------- ------------------- --------\n",
    "\n",
    "Depending on which definition we choose, the data might provide evidence\n",
    "for or against the hypothesis that the coin is biased, but in either\n",
    "case it is relatively weak evidence.\n",
    "\n",
    "In summary, we can use Bayesian hypothesis testing to compare the\n",
    "likelihood of $F$ and $B$, but we have to do some work to specify\n",
    "precisely what $B$ means. This specification depends on background\n",
    "information about coins and their behavior when spun, so people could\n",
    "reasonably disagree about the right definition.\n",
    "\n",
    "My presentation of this example follows David MacKay’s discussion, and\n",
    "comes to the same conclusion. You can download the code I used in this\n",
    "chapter from <http://thinkbayes.com/euro3.py>. For more information see\n",
    "Section [download]."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Discussion\n",
    "\n",
    "The Bayes factor for `B_uniform` is 0.47, which means that the data\n",
    "provide evidence against this hypothesis, compared to $F$. In the\n",
    "previous section I characterized this evidence as “weak,” but didn’t say\n",
    "why.\n",
    "\n",
    "Part of the answer is historical. Harold Jeffreys, an early proponent of\n",
    "Bayesian statistics, suggested a scale for interpreting Bayes factors:\n",
    "\n",
    "    ---------- -------------------------\n",
    "    Bayes      Strength\n",
    "    Factor     \n",
    "    1 – 3      Barely worth mentioning\n",
    "    3 – 10     Substantial\n",
    "    10 – 30    Strong\n",
    "    30 – 100   Very strong\n",
    "    $>$ 100    Decisive\n",
    "    ---------- -------------------------\n",
    "\n",
    "In the example, the Bayes factor is 0.47 in favor of `B_uniform`, so it\n",
    "is 2.1 in favor is $F$, which Jeffreys would consider “barely worth\n",
    "mentioning.” Other authors have suggested variations on the wording. To\n",
    "avoid arguing about adjectives, we could think about odds instead.\n",
    "\n",
    "If your prior odds are 1:1, and you see evidence with Bayes factor 2,\n",
    "your posterior odds are 2:1. In terms of probability, the data changed\n",
    "your degree of belief from 50% to 66%. For most real world problems,\n",
    "that change would be small relative to modeling errors and other sources\n",
    "of uncertainty.\n",
    "\n",
    "On the other hand, if you had seen evidence with Bayes factor 100, your\n",
    "posterior odds would be 100:1 or more than 99%. Whether or not you agree\n",
    "that such evidence is “decisive,” it is certainly strong."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Exercises\n",
    "\n",
    "Some people believe in the existence of extra-sensory perception (ESP);\n",
    "for example, the ability of some people to guess the value of an unseen\n",
    "playing card with probability better than chance.\n",
    "\n",
    "What is your prior degree of belief in this kind of ESP? Do you think it\n",
    "is as likely to exist as not? Or are you more skeptical about it? Write\n",
    "down your prior odds.\n",
    "\n",
    "Now compute the strength of the evidence it would take to convince you\n",
    "that ESP is at least 50% likely to exist. What Bayes factor would be\n",
    "needed to make you 90% sure that ESP exists?\n",
    "\n",
    "Suppose that your answer to the previous question is 1000; that is,\n",
    "evidence with Bayes factor 1000 in favor of ESP would be sufficient to\n",
    "change your mind.\n",
    "\n",
    "Now suppose that you read a paper in a respectable peer-reviewed\n",
    "scientific journal that presents evidence with Bayes factor 1000 in\n",
    "favor of ESP. Would that change your mind?\n",
    "\n",
    "If not, how do you resolve the apparent contradiction? You might find it\n",
    "helpful to read about David Hume’s article, “Of Miracles,” at\n",
    "<http://en.wikipedia.org/wiki/Of_Miracles>."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.5.1"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}
